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ABSTRACT 

A sample research observation measure a nd its 
preparation into a written observer training package for use in 
evaluating children within or across classrooms is described. The 
Planned Activity Check (PLA-Check) measure, a time-sample observation 
of children's participation is planned activities; was used, A 
training manual based upon the PLA-Check was devised and then revised 
on the basis of a series of user tests. The use of the PLA--Check 
training manual was found to be sufficient as an exportable training 
program for observers on the use of a standardized measurement system 
of classroom participation. The study also provides an example of 
procedures useful in developing a measure for general use. Last, the 
PLA-Check as a measure provides an evaluation process for across 
setting comparison of classrooms and techniques. Tables and graphs 
are included in the study to show the reliability of the observer at 
different phases of development. (MP) 
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Development of a Standardized 
f^easure of Classroom Participation' 



Michael F. Cataldo and Todd R. Risley 
University of Kansas 



The number of program erploylng applied behavior analysis technlqucjs Is Increasing 
to the extent that an evaluation and conparlson of these programs would be desirable* 
The common use of achievement tests Is not particularly suitable for such cornparlson 
because most such tests assess an Individual student on specific subject content. 
By contrast, hehavioral programs are most often Instituted for an entire class rather 
than as separate programs for each student. The need, then. Is for a behavioral 
measure which provides an overviev; of classroom techniques. 

CS^ln addition to behavioral scientists and educators, the Federal Government also seems 
to be Interested in this issue. A case in point is a 1071 report (HPJ, 1971) to 

rH the Secretary of Health, Education, and 'Welfare v/hich recommended evaluation of the 
behavioral techniques employed in both special and regular classrooms. Specifically, 

*^xp the report recommended that standardized procedures for the classroom evaluations 
be developed from the measures used in behavioral research. 

CO It should be clear, and the previous papers in this session provide a good example, 
>^ that we have an extensive technology of behavioral measures for use in the classroom. 
^ The question now is how to standardize sone of these research measures in convenient 
written form so that they nay be reliably employed in general classroom evaluations. 

We undertook an exercise to find out if it would be possible to accomplish this 
easily; that is, vie endeavored to take a relatively simple research observation 
measure and go throu^ih the steps to prepare it into a written observer training packi^e 
^ suitable for general use in evaluating groups of children in classrooms as a whole 
and for comparing different classrooms. 

The measure we chose has been reported in previous research as the Planned Activity 
Check--or PLA-Check (Hoke and Risley, 1971; LeLaurin and Risley, 1972; f)oke and 
Risley. 1972; Risley and Cataldo, in pressK In these studies, the PLA-Check has 
been used to Investigate the effects of various schedules, routines, activities* types 
of materials, and methods of presenting materials. 

The PLA-Check measure is a time-sample observation of children's participation in 
planned activities- Let me explain briefly the procedures for using the PLA-Check 
measure. 

On the day preceding the evaluation, the teacher is asked to list the activities she 
has plBinned for the next day. She is also asked to specify those student behaviors 
that define appropriate use of materials and/or participation in planned activities. 
Throughout the day of the evaluation, time-sample observations are made of the number 
of students present and the number participating. This procedure provides Information 
on the pattern of participation during the day and ansv/ers the question of ho^/well 
plans were made by t!ie teacher and ho^^ well these plans v/ere implemented. 
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As a first step, we devised a training manual based upon the PLA^Check procedures 
used in previous research studies. The training manual describing the use of the 
PLA-Check measure was developed to provide three levels of information: 

General information about how and where the measure may be used, with 
examples of^its use in previous evaluations. 

Suj)ervisor's infomation on the use of the measure, the materials 
neec^ed, schedulinq of observers aiid observations, the computation 
and use of data and reliability. 

Observer training information of step-by-step procedures on taking 
obser^afions, use of recordinn forms, and computation of data. 

Me then subjected this manual to a series of user tests and revised it based upon 
the results of Its use. As v^e proceeded, each successive phase of investigation 
more closely approximated the PLA-Check procedures by the target consumer: education 
prograns of various types at different locations. Inter-observer reliability v^as 
used as the dependent variable to indicate the need for improvement the niaterials 
and to measure the project's progress tov/ard a readily usable and reliable measure- 
ment procedure. Development and testing of the draining manual proceeded in three 
phases. 

Phase 1 . In this phase, the complete training manual v/as revised until five pairs 
of nafve observers could be exposed only to the training manual and obtain point-by- 
point reliability with each other above the 00 percent level on a sample class observa- 
tion. The results of this phase are presented in Figure 1* Each data point represents 
a different pair of naive observers' reliability on a sample classroom observation. 
Sample observations ranged from 15 to 44 minutes, depending upon the length of the 
activity period observed. As indicated in Figure 1, four revisions were made resulting 
in a training manual sufficient to Instruct rraive observers to attain 90 percent 
reliability in actual classroom observation. 

Phase II . Five naive observer pairs received instruction on use of the measure solely 
from the training manual. Each observer pair v/as then assigned to collect data on an 
entire class day. Each ;.^air was assigned to one of five different grade levels: a 
preschool, a first, second, and third nrade, and a high school class. 

The results of Phase II are presented in Figure 2 , and shoi*; that the mean reliability 
on observations in each class v^as high: in one class, the third grade, observers 
obtained 87 percent agreement; and, in the other four classes. Inter-observer agreement 
was above 90 percent. 

Phase III . In the final phase, five directors of classroom projects in other loca- 
tions v/ere subcontracted to obtain a full class day's data on one of their classrooms » 
using only the training manual with two observers selected by thein. The classes 
' observed v^ere: a community-sponsored preschool, a Behavior Ar^alysis Foliar-Through 
kindergarten, a contingency-management third grade, a combined open-classroom fifth 
grade, and a traditional public school seventh grade. 

The Phase III results presented in Figure 3 demonstrate that the mean reliability 
of the observers selected by directors of other education programs was above P>5 percent 
for each class day observed; and four of the five locations demonstrated above 90 
percent observer agreement. 
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Also dunnn Phase III, one of the experienced observers at the University of Kansas 
who had participated in the development of the traininq materials travelled to each 
location and made independent, simultaneous observations of the classrooms v/ith the 
local observers. Figure 4 shov/s that the me^n reliability scores for observations 
made by the developers' observers and one or both of the consumers* observers v/as 
above 85 percent for each class observed. 

Data obtained during the development and testing of the PLA-Check training materials 
also sho»//ed that high inter-observer agreement \'Jf\s obtained v^hether class participation 
was high or lov*/. 

Table 1 indicates that inter-observer reliability on observations in preschool through 
college classes ranned from Sfl to 97 percent, even though the student participation in 
these classes ranged from 42 to ^0 percent. 

P' more detailed look at oarticipation and reliability is presented in Figure 5 , In 
this figure, data from a preschool and a third-grade class viere analyzed. The preschool 
represents an example of a poor class with participation averaging 50% or lov/er. The 
thir'^ grade represents a class with hinh participation, usually above 70%. In both 
classes across successive days, reliability was consistently above PS%. Thus, reli- 
ability vms consistently high regardless if the class had generally low participation 
or high participation or on occasion varied substantially from previous levels, as in 
the case of days 13 and 14 in the third-grade class. 

Participation and reliability data for the preschool and third ^rade are presented 
for individual PLA-Checks across an entire class day's activities in Figure 6 . Again, 
the data show that, even at the level of individual checks, reliability is high whether 
the class has high, lov/, or variable levels of participation. 

In conclusion, v/e'd like to make three points about this demonstration. 

First, the results show the PLA-Check training manual to be sufficient as an exportable 
training program for observers on the use of a standardized measurement system of 
classroom participation. 

Second, the study also provides an example of procedures useful in developing a 
measure for general use. These procedures include the use of naive observers in a 
variety of educational settings with continued revision until training materials are 
adequately described to be sufficient to direct people's behavior. 

Last, the PLA-Check as a measure provides an evaluation process for across-setting 
comparison of classrooms and techniques. The measure also provides convenient proce- 
dures for other investigators to conduct further classroom research, curriculum 
development, teacher traininn, and design of teacher activities. 
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Figure 1. Scatter diagram of reliability scores of Phase I of the 
study In which 26 different observer-pairs each employed 
one of four versions of the PLA-Check Training Manual • 
Each point represents the percentage of agreement between 
one observer pair making a PLA-Check observation of a class.. 
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Reliability of Manual- trained 
observers selected by 
developers 
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gure 2. Mean and range of Inter-observer reliablHty scores for 
Phase II in which observer pairs employing the PLA-Check 
Training Manual obtained a representative class day's 
data of five different grade levels. Reliability scores 
for each setting represent observation by a different pair 
of observers. 



Reliability of Manual- trained 
observers selected by five 
different consumers 
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Figure 3. Phase III meaii and range reliability scores of observer- 
pairs chosen by five classroom research program directors 
subcontracted to obtain a representative day's partici- 
pation data using the PLA-Check Training Manual . 
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Reliability between developers' 
observers and consumers' 
Manual-trained observers 
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Figure 4. Mean and range of reliability between a trained observer 
of the Training Manual Development Staff and one or both 
of the observers at each of the subcontract sites. The 
data represents at least 30 PLA-Check counts at each of the 
sites. 




Figure 5, Participation and reliability between observers 1n a preschool 
where participation levels are generally low and in a third- 
grade c'tass room where participation is high. 
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Figure 6. Participation and reliability between observers through an 
entire class day in a preschool and a third-grade classroom. 
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Reliability Scores of Observe? Pairs 

for Three Observations Using Version IV 

of the PLA-Check Training Manual 
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1. Inter-observer reliability on observations in preschool 
through college classes with a wide range of student 
participation levels. 



